Papers with latency reduction
Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement (2026.eacl-industry)
Copied to clipboard
Aaditya Shukla, Sidney Knowles, Meenakshi Madugula, David Farris, Ryan Angilly, Santiago Pombo, Lu An, Anbang Xu, Abhinav Balasubramanian, Tan Yu, Jiaxiang Ren, Rama Akkiraju
| Challenge: | NVInfo AI is a generative AI agent that can be deployed in production without full-scale retraining or infrastructure overhauls. |
| Approach: | They propose to implement a retrieval-augmented generation (RAG)-driven data flywheel in NVInfo AI, a mixture-of-experts knowledge assistant, for 30,000 employees. |
| Outcome: | The proposed system addresses failures in retrieval-augmented generation pipelines and enables continuous learning. |
ATOM: AdapTive and OptiMized dynamic temporal knowledge graph construction using LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Unstructured data is expanding at an unprecedented rate, and static knowledge graphs are often overlooked due to their dynamic nature and lack of time-sensitive features. |
| Approach: | They propose a few-shot approach that builds and continuously updates Temporal Knowledge Graphs (TKGs) from unstructured texts. |
| Outcome: | Empirical results show that ATOM achieves 18% higher exhaustivity, 33% better stability, and over 90% latency reduction compared to baseline methods. |
SLiM: Speculative Decoding with Hypothesis Reduction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Speculative decoding has emerged as an alternative to autoregressive decoding for expediting inference in large language models (LLMs). prevailing assumptions focus solely on latency reduction, neglecting the computational expenses. |
| Approach: | They propose a speculative decoding enhancement to reduce the speculation set while validating more effective tokens. |
| Outcome: | The proposed method reduces the speculation set while validating more effective tokens. |
Online Semantic Parsing for Latency Reduction in Task-Oriented Dialogue (2022.acl-long)
Copied to clipboard
| Challenge: | Standard conversational semantic parsing maps a user's intent into an executable program, but execution is slow when expensive function calls are included. |
| Approach: | They propose a task of online semantic parsing to predict and execute function calls while the user is still speaking. |
| Outcome: | The proposed approach reduces latency with good parsing quality and execution cost. |
SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget (2024.acl-long)
Copied to clipboard
| Challenge: | Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) but memory-constrained devices are a major concern in edge AI training and serving. |
| Approach: | They propose a framework for efficient serving of MoE-based large language models with tunable memory budgets. |
| Outcome: | Experiments show that SwapMoE can reduce memory consumption while maintaining reasonable accuracy. |
Energy Considerations of Large Language Model Inference and Efficiency Optimizations (2025.acl-long)
Copied to clipboard
| Challenge: | Prior benchmarking efforts focused on latency reduction in idealized settings, often overlooking real-world inference workloads that shape energy use. |
| Approach: | They propose a modeling approach that approximates real-world LLM workflows . they show that the effectiveness of inference optimizations is sensitive to workload geometry . |
| Outcome: | The proposed approach reduces energy use by 73% from unoptimized baselines. |